Contact lens system

ABSTRACT

A contact lens system (110) for placing in an eye for augmented reality systems is disclosed. The contact lens system (110) comprises a display (111) comprising a matrix of display elements; a driver unit (112) configured to receive data from a host (120) and to present the data on the display (111) and a set of sensors (113) integrated on the display (111) for measuring pupil size. The driver unit (112) is further configured to read outputs from the set of sensors to determine the pupil size and adjust a size of an active area of the display.

CROSS REFERENCE TO RELATED APPLICATION(S)

This application is a continuation of U.S. application Ser. No.17/598,073, filed Sep. 24, 2021, which is a 35 U.S.C. § 371 NationalPhase Entry Application from PCT/EP2019/057621, filed Mar. 26, 2019,designating the United States. The disclosures of the referencedapplications are incorporated herein in their entirety by reference.

TECHNICAL FIELD

Embodiments herein relate to a contact lens system and method therein.In particular, they relate to a contact lens system with dynamic activearea display for augmented reality systems.

BACKGROUND

Displays in augmented reality systems are evolving fast. Research hasbeen done in the fields of adding electronics on a contact lens. Lenselectronics have been used for multiple applications, such as dynamicvision correction or displaying images. In R. Blum et. al, “Enhancedelectro-active lens system,” 2007, electro-active vision correction hasbeen proposed as means of adjusting lens correction depending ondifferent conditions, such as distance from objects being viewed,ambient light or size of the pupil. In B. A. PARVIZ, “Augmented Realityin a Contact Lens,” IEEE, 2010, researchers at University of Washingtonmade a contact lens with 64 pixels that was tested on rabbits.

The pupil dilates, i.e., have different sizes, at differentillumination, state of mind, distance to the object in focus, etc. Pupilsize does not directly affect the field of vision, but it does affectthe perceived depth-of-field, such that objects will appear blurrier inthe edges of the vision with smaller pupil size, as described in S.Marcos, et. al., “The depth-of-field of the human eye from objective andsubjective measurements,” Vision Res., 1999.

Things are different for objects displayed very close to the eyes, e.g.,in a contact lens display. If the pupil size is smaller than the displaysize, then light from the edges of the display will not be perceived.That means that a display on a contact lens may be too small for thepupil at one scenario and too large in another.

US2014/0240665A1 discloses an eye-facing pupil diameter sensing systemfor an ophthalmic lens comprising an electronic system. The eye-facingpupil diameter sensing system is utilized to determine pupil diameterand use this information to control various aspects of the ophthalmiclens. The pupil diameter sensor is implemented as an array of smallersensors placed at various locations in the contact lens to samplevarious points on the iris. Sensors may determine pupil diameter andchanges thereof by detecting light reflection, impedance,electromagnetic field, neural activity, muscle activity, and otherparameters as are known in the ophthalmic art. The possibility ofincorporating an image display into the lens is briefly mentioned.

SUMMARY

It is therefore an object of embodiments herein to provide an improvedcontact lens system and method to counter act the problem describedabove.

According to one aspect of embodiments herein, the object is achieved bya contact lens system for placing in an eye. The contact lens systemcomprises a display comprising a matrix of display elements; a driverunit configured to receive data from a remote host and to present thedata on the display; and a set of sensors integrated on the display formeasuring pupil size. The driver unit is further configured to readoutputs from the set of sensors to determine the pupil size and adjust asize of an active area of the display based on the pupil size byactivating and deactivating the display elements.

According to one aspect of embodiments herein, the object is achieved bya method performed in a contact lens system for placing in an eye. Thecontact lens system comprises a display comprising a matrix of displayelements; a driver unit configured to receive data from a remote hostand to present the data on the display; and a set of sensors integratedon the display for measuring pupil size. The driver unit reads outputsfrom the set of sensors to determine the pupil size and adjust a size ofan active area of the display based on the pupil size by activating anddeactivating the display elements.

In other words, the contact lens system and method therein according toembodiments herein enables the size of the display being dynamicallyadjusted according to the pupil size. By measuring the pupil size, thedisplay may dynamically change in size so that maximum amount of pixelsthat can be perceived by the user, can be used to display content.

Some advantages the contact lens system according to embodiments hereinare that the user will always be able to see the maximum size of thedisplay and that the data to be presented on the display may bedynamically adjusted based on the size of active area of the display.

Therefore, the embodiments herein provide an improved contact lenssystem and method for augmented reality systems.

BRIEF DESCRIPTION OF THE DRAWINGS

Examples of embodiments herein are described in more detail withreference to attached drawings in which:

FIG. 1 is a schematic block diagram illustrating an augmented realitysystem in which a contact lens system according to embodiments hereinmay be implemented;

FIG. 2 is a schematic showing a contact lens system with differentdisplay sizes and photo sensors according to one embodiment herein;

FIG. 3 is a schematic showing a contact lens system with differentdisplay sizes and photo sensors according to another embodiment herein;

FIG. 4 is a schematic showing a contact lens system where sensors arepositioned by each sensor replacing one or more display pixels;

FIG. 5 is a schematic showing a contact lens system where sensors arepositioned in between rows and columns of the display elements;

FIG. 6 is a flow chart illustrating a method performed in a contact lenssystem according to embodiments herein;

FIG. 7 is a schematic showing a simplified mechanism of how to use photodiodes to detect the pupil size.

FIG. 8 is a diagram illustrating responses of photo sensor on an LTPSsubstrate; and

FIG. 9 is a diagram illustrating activation curves for a contact lenssystem with different numbers of sensors according to embodimentsherein.

DETAILED DESCRIPTION

FIG. 1 depicts a block diagram of an augmented reality system 100 inwhich a contact lens system 110 according to embodiments herein may beimplemented. The augmented reality system 100 comprises a contact lenssystem 110 for placing in an eye and a host 120 that sends informationand interacts with the contact lens system 110. The contact lens system110 comprises a display 111 comprising a matrix of display elements, adriver unit 112 configured to receive data from the remote host 120 andto present the data on the display 111, a set of sensors 113 formeasuring the pupil size. The set of sensors 113 may be integrated onthe display 111. The driver unit 112 controls what is outputted on thedisplay 111 and handles the input from the set of sensors 113. The Host120 interacts with the contact lens system 110 via the driver unit 112.The display 111 with the matrix of display elements is positioned on thecontact lens and is able to emit light in a structured way.

The contact lens system may comprise a pair of contact lenses, one lensfor each eye of the user. The lenses may be identical and display thesame or different data.

The set of sensors 113 may be photodiodes that measures the lightreflected from the eye. Other types of pupil size or diameter sensorsmay also be used. For example, as disclosed in US2014/0240665A1, the setof sensors may be a single- or multi-turn coil antenna. Such an antennamay receive electromagnetic radiation from the eye as the musclescontrolling the iris contract and relax. It is well-known in therelevant art that muscle and neural activity of the eye may be detectedthrough changes in electromagnetic emissions, for example with contactelectrodes, capacitive sensors, and antennas. In this manner, a pupildiameter sensor based on a muscle sensor may be implemented. The pupildiameter sensor may also be implemented as one or more contact- orcapacitive electrodes designed to measure impedance across the eye.Impedance may be used to detect changes in pupil diameter. For example,the impedance measured across the iris and pupil may change appreciablydepending on pupil diameter. A pupil diameter sensor placed at theappropriate location on the eye and properly coupled to the eye coulddetect these changes in impedance and hence the pupil diameter or size.However, not all of these sensors can be incorporated into the display.Some of these may be placed outside the display, as shown in FIG. 1sensors 113 with dotted line.

The display 111 may have at least two display sizes defined by a matrixwith different numbers of rows and columns of the display elements. Eachdisplay size is herein referred to as a valid active area setting of thedisplay. So an active area of the display may be adjusted to havedifferent sizes, where the numbers of rows and columns of the displayelements which are active for different display sizes are different. Thedisplay elements may herein also be referred to as pixels. The amount ofavailable display sizes varies from minimum of two, i.e. minimum andmaximum, and up to the number of pixels on the diagonal of the displaydivided by two, if all pixel row-column interceptions have a sensor. Butthere is no reason of having a smaller active area than the smallestpupil dimension. Normally the pupil ranges from 2 mm-8 mm in diameter.The display 111 and its active areas may be in different shapes, where arectangular shape and a round shape are most common for a contact lens.

FIG. 2 shows one example of a contact lens 110 with different displaysizes in rectangular shapes and a set of sensors 113. As shown in FIG. 2, the set of sensors may be positioned diagonally in a corner within thedisplay 111 at an edge of each display size. The set of photo sensorsmay be any format that is supported by the display 111.

FIG. 3 shows another example of a contact lens 110 with differentdisplay sizes in a concentric pixel arrangement and a set of sensors113. As shown in FIG. 3 , the set of sensors may be positioned atdifferent places within the display at an edge of each display size.

For both FIGS. 2 and 3 , all four “corners” of possible active areas maybe used. Then it will be less obvious where the sensors are positionedfrom the user's perspective. This will give a better user experiencethat is less perceived disturbance for the user.

The sensors 113 shown in FIGS. 2 and 3 may be photo sensors. Each photosensor is designed to look into the eye and are shielded from lightcoming from the outside, thus only activated by light that is reflectedin the eye. Measurements are performed when the display elements orpixels are in off state to avoid crosstalk the measurement from thedisplay.

The set of photo sensors may be positioned at intersections of the rowsand columns of the display elements matrix, as shown in FIG. 4 . Thatis, they may be positioned by each sensor replacing one or more displayelements or pixels. Then the places occupied by the sensors will appearas dead pixels. When replacing one or more display elements or pixels,the photo sensors may be distributed in different arrangements, as longas they are in an intersection of a row and column, for example in thecorners of each display size.

It is also conceivable to position the set of sensors in between rowsand columns of the display elements, as shown in FIG. 5 . Then the rowsand columns of the display elements will stay intact and places occupiedby the sensors will not appear as dead pixels, but the sensors stillmight be perceived as such if they are large enough.

According to the embodiments herein, the driver unit 112 is furtherconfigured to read outputs from the set of sensors 113 to determine thepupil size and adjust a size of an active area of the display 111 basedon the pupil size by activating and deactivating the display elements.

According to some embodiments herein, the contact lens system 110 mayfurther comprise one or more IR diodes placed close to a sensor toprovide light towards the eye to reflect.

In the following a method performed in the contact lens system 110 foradjusting a size of an active area of the display 111 will be describedin detail with reference to FIG. 6 . As described above, the contactlens system 110 comprises a display 111 comprising a matrix of displayelements, a driver unit 112 configured to receive data from a host 120and to present the data on the display 111 and a set of sensors 113integrated on the display 111 for measuring pupil size. The methodcomprises the following actions, which actions may be performed in anysuitable order.

Action 610

Before each new frame to be presented on the display 111 or at regularintervals, e.g. every tenth frame, a few times per second or everysecond, the driver unit 112 checks the set of sensors 113 in order todetect the pupil size. Even if the processing and reading of the sensorsignals only consume low power, a less frequent check will save somepower. The driver unit 112 reads outputs from the set of sensors 113.The photo sensors are directed into the eye to be able to distinguishthe edge between the pupil and the iris. Depending on the distancebetween the photo sensors and the number of photo sensors, the responsecurves when reading out the photo sensors will differ. See descriptionof FIG. 9 below.

In some cases, the light from the display elements may be used as thelight source for reflectance measurements, i.e. in dark environments.This is to get the best possible signal to noise ration (SNR) to be ableto find a threshold for the reflected light for the set of sensors.

In one embodiment, the driver unit 112 may activate the relevant displayelements to provide light towards the eye to reflect. For example, thedisplay elements surrounding one photo sensor may be lighted up and thereflected rays from the display elements will be measured by the photodiode. Suitably the display elements outside the current pupil sizeshould be activated to be able to find the edge. Further the displayelements outside the current active area of the display may also beactivated to provide light.

The driver unit 112 may activate the relevant display elements one at atime or all at once, i.e. the relevant display elements may be flashedquickly at once or may be activated one by one in a sweeping manner.

In one embodiment, one or more infrared (IR) diodes may be placed closeto a sensor to provide light towards the eye to reflect. One or severaladditional IR diodes may be placed next to or close to any one or all ofthe photo sensors. The direction of the light should be towards the eye.By adding the IR diode there is no need for surrounding illumination tobe able to detect the pupil size. The reflected IR rays from the IRdiodes are measured instead. The benefit is that it results a morestable measurement as it is not dependent on enough light from thesurroundings. The down part is if at least one more pixel area for thedisplay is exchanged for an IR diode, this leaves less space for pixelson the display.

Action 620

The driver unit 112 determines the pupil size based on the outputs fromthe set of sensors.

Depending on the pupil diameter, sensors at various distances from thecenter of the iris will detect different amounts of reflected light. Forexample, when the iris is dilated most of the sensors may detect littlelight because of the large, dark pupil. Conversely, when the iris isconstricted most sensors may detect higher light because of reflectionof the iris.

FIG. 7 shows a simplified mechanism how to use photo diodes to detectthe pupil size. The driver unit 112 may detect the edge between thepupil and iris by comparing the detected light from the set of sensors.In the pupil area, there is almost no light reflection, so photo diodes701, 702 will not detect any reflected light and no output signal. Photodiodes 703, 704, 705 . . . will detect reflected light and have outputsignal. By comparing the detected light from the set of photo diodes701, 702, 703, 704 . . . the edge between the pupil and iris may befound and the display size is selected based on this.

The driver unit 112 may determine the size of pupil by comparing thereflected light with a threshold. The granularity of the sensors ispredefined or configured, i.e. the position, the distance between eachsensor and how many sensors per area. The number of photo sensors thatget reflected light enough to have an output signal over the thresholdis counted and from that the pupil size is decided and the display sizeis selected based on this.

The difference in signal level from the photo sensors will be dependenton whether the light that is reflected by the iris or by the inner areasurrounded by the iris, i.e. the pupil, with hardly any reflection.Differences may also arise with an iris of different color and resultingdifferent reflectance at different areas of the iris and betweendifferent users.

So the light reflected will depend on the color of the iris and ofcourse on the amount of light emitted from the environment and from thedisplay. Therefore the threshold should be adjusted or calibration maybe needed to take account these variations. The threshold for reflectedlight may be adjusted depending on the reflectance or brightness of theiris.

A calibration process may be implemented where the host 120 or thedriver unit 112 may control the light output of the contact lens display111. The gain of the photo sensors may also be controlled to find asuitable level of sensitivity. If no edge is detected between the pupiland iris, threshold can be lowered, while if multiple edges aredetected, threshold can be increased.

There might be problems if the iris is dark brown and the lightconditions of the environment and the display are low. Then the lightreflected might be too little. Turning on display elements shortlybefore measuring or using IR diodes may solve this problem.

Photo sensors on a Low-temperature polycrystalline silicon (LTPS)substrate may be used which have the wavelength sensitivity as shown inFIG. 8 , according to the article of W. J. Chiang et. al., “Siliconnanocrystal-based photosensor on low-temperature polycrystalline-siliconpanels”, Applied Physics Letter, 91, 051120, 2007. FIG. 8 shows lightabsorption spectrum of samples with silicon nanocrystal layers of 100,200, and 300 nm thickness monochromatically illuminated by a constantoptical power of 35 W/cm² and a bias voltage of 2.5 V. Other substratesthat can be used may be Amorphous Silicon (A-Si), Indium Gallium ZincOxide (IGZO) or similar technologies.

Action 630

The driver unit 112 determines if the pupil size is the same as from theprevious measurement. If the pupil has not changed size, the driver unit112 will check and read outputs from the set of sensors 113 again. Ifthe pupil has changed size, the driver unit 112 will adjust the size ofan active area of the display based on the pupil size by activating anddeactivating the display elements.

So once the pupil size is determined based on the threshold and thegranularity of the photo sensors, the active area of the display mayneed to be decided and adjusted. That is appropriate number of rows andcolumns of the display elements may be activated based on the pupilsize. Depending on the number of photo sensors placed in the display111, the precision of the active area size will differ. FIG. 9 shows anexample of activation curves where 3, 6 and 18 photo sensors are used,where the y axis is the pupil size ranging from 2 mm to 8 mm, and the xaxis is the number of photo sensors included in the set of sensorswithin the active area that ranges from 2 mm to 8 mm, indicated as theamount of possible display sizes. It can be seen from FIG. 9 , that lessphoto sensors will give a jagged activation curve, i.e. when using fewerphoto sensors, the adjusting of the display active area will have largerdiscrete steps.

Action 640

Once the size of the active area is decided based on the threshold andthe granularity of the photo sensors, the new active area size needs tobe communicated. So according to some embodiments herein, the driverunit 112 may send the size of the active area of the display to the host120.

There are at least two different ways to handle the different activearea sizes of the display.

According to some embodiments herein, the driver unit 112 eithertransmits the new display size to the host 120 or the host 120 asks ifthere is a new size and retrieves the new size from registers in thedriver unit 112. The host 120 then adjusts its internal display buffersand settings so that a correct size of image is sent to the driver unit112 in the contact lens 110. If the host can adjust the image size,power will be saved in the whole system 100 as the image reduction isdone already on the host.

Action 650

According to some embodiments herein, the driver unit 112 may alwaysreceive the same size of image and depending on the active area sizethat is decided based on the photo sensors, the driver unit 112 willreduce the size of the image accordingly. This is a more flexiblesolution which will give the host 120 more freedom. The amount of datahandled in the host 120 and that is transferred to the driver unit 112is the same (maximum) size independent on the active area size of thedisplay, thus consuming more power.

So according to some embodiments herein, the driver unit 112 may beconfigured to adjust frame buffer size of an image received from a hostbased on the size of active area of the display.

Action 660

The pupil size will be affected by changes in the amount of lightentering the eye. In order not to change the pupil size when theluminance is changed on the lens display due to the change of thedisplay size or other reasons, there is a control that keeps the sameamount of lumen output from the display. So when the active area of thedisplay gets larger the luminance of the display will drop so that thesame amount of lumen will hit the retina. The opposite will happen whenactive display area gets smaller, the luminance of the display willincrease to keep a consistent lumen.

So the total luminance out of all active display elements should be keptat or not exceeding a defined value, e.g. a maximum lumen, max_lumen, asexpressed in Eq.1:Σlumen_(pixel)=max_lumen  Eq. 1

That is the maximum lumen out of the display should always be the sameindependent on how many active display elements. For AR applications,the size of the virtual objects is important to consider. Displayinglarge virtual objects might produce lots of lumens, while displayingshort texts and bounding boxes might require less. Therefore the maximumlumen output should depend on that too. The defined lumen value may be apreferred value or automatic adjusted according to likings of individualusers. This is done to have a more comfortable usage and the user shouldexperience the same light intensity to the retina.

Therefore according to some embodiments herein, the driver unit 112 maybe configured to adjust luminance of the active display elements to keepthe total amount of lumen output from the display at a defined value.The driver unit 112 may be further configured to adjust the definedvalue according to preferred settings of individual users.

To summarize, the contact lens system 110 and method therein accordingto the embodiments herein provide an improved visual system foraugmented reality systems with at least the following advantages:

Eye-mounted displays bring significant advantages in comparison tonon-eye-mounted displays.

From the usability perspective, contact lenses could be promising aswearable displays, since they are naturally discrete, and the formfactor is widely accepted. Moreover, the amount of energy required todrive the displays would be significantly less than displays that arefarther away.

By measuring the pupil size, the display can dynamically be changed insize so that maximum amount of pixels can be used to display content.

The user will always be able to see the maximum size of the display andthat the data needed is dynamically adjusted based on the active displaysize.

Integrating the display and the sensors in a unique way by manufacturingthe sensors in the same process steps as the display elements simplifiesthe manufacture.

The system always provides a display with maximum field of view,regardless of changes to the pupil size, and without wasting power onpixels that are outside the field of view. The system saves power byreducing the display size and turn off pixels in the periphery, whichwould not be able to be seen anyway since they would be out of the fieldof view of the eye. When the pupil size increase, then the display sizecan get larger again to fill the field of view.

The system increases the user experience by maintaining perceivedbrightness of the display constant or at a preferred/comfortable levelby adjusting the display luminance based on the display size. This canalso be seen as an optimization on power usage.

There is no need for additional hardware to adjust the size of theactive area of the display. The contact lens system itself includes allneeded hardware to accomplish this.

When using the word “comprise” or “comprising” it shall be interpretedas non-limiting, i.e. meaning “consist at least of”.

The embodiments herein are not limited to the above described preferredembodiments. Various alternatives, modifications and equivalents may beused. Therefore, the above embodiments should not be taken as limitingthe scope of the invention, which is defined by the appended claims.

The invention claimed is:
 1. A contact lens comprising: a displaycomprising display elements; a driver configured to present informationon the display; and one or more sensors configured to measure a pupilsize of an eye to which the contact lens is attached; wherein the driveris further configured to adjust a size of an active area of the displaybased on the measured pupil size so that only display elements which canbe perceived by the eye present information.
 2. The contact lensaccording to claim 1, wherein the display elements are arranged in rowsand columns and the active area is adjusted to have at least two sizesdefined by different numbers of the rows and columns of the displayelements.
 3. The contact lens according to claim 2, wherein the one ormore sensors are positioned diagonally within the display at an edge ofeach active area.
 4. The contact lens according to claim 2, wherein theone or more sensors are positioned at different places within thedisplay at an edge of each active area.
 5. The contact lens according toclaim 2, wherein the one or more sensors are positioned at intersectionsof the rows and columns of the display elements.
 6. The contact lensaccording to claim 1, wherein the one or more sensors are positioned inbetween the display elements.
 7. The contact lens according to claim 1,wherein the one or more sensors are photodiodes configured to detectlight reflected from the eye.
 8. The contact lens according to claim 7,wherein the driver is further configured to determine the size of thepupil by comparing an output signal for the photodiodes that detectedthe reflected light with a threshold.
 9. The contact lens according toclaim 8, wherein the driver is further configured to adjust thethreshold based on the reflected light or brightness from an iris of theeye.
 10. The contact lens according to claim 1, wherein the driver isfurther configured to adjust a luminance of the display elements of theactive area to keep the total amount of lumen output from the display ator below a defined value.
 11. The contact lens according to claim 10,wherein the driver is further configured to adjust the defined valueaccording to user settings.
 12. The contact lens according to claim 1,wherein the driver is further configured to use the display elements byactivating selected display elements to provide light towards the eyefor reflection.
 13. The contact lens according to claim 1, furthercomprising one or more infrared (IR) diodes to provide light towards theeye.
 14. The contact lens according to claim 1, wherein the driver isfurther configured for interaction with a host, wherein the interactionincludes receiving information from the host.
 15. The contact lensaccording to claim 1, wherein the driver is further configured to adjusta size of an image for presentation on the display based on the size ofthe active area of the display.
 16. A method for adjusting a size of anactive area of a display in a contact lens, wherein the displaycomprises display elements and one or more sensors, the methodcomprising: using the one or more sensors to measure a pupil size of aneye to which the contact lens is attached; and adjusting a size of anactive area of the display based on the measured pupil size so that onlydisplay elements which can be perceived by the eye present information.17. The method according to claim 16, further comprising interactingwith a host, wherein the interacting includes receiving information fromthe host.
 18. The method according to claim 16, further comprisingadjusting a size of an image for presentation on the display based onthe size of the active area of the display.
 19. The method according toclaim 16, further comprising adjusting a luminance of the displayelements of the active area to keep the total amount of lumen outputfrom the display at or below a defined value.